Hybrid registration method

ABSTRACT

A registration method whereby a sensor-based approach is used to establish initial registration and whereby upon the commencement of navigating an endoscope, image-based registration methods are used in order to more accurately maintain the registration between the endoscope location and previously-acquired images. A six-degree-of-freedom location sensor is placed on the probe in order to reduce the number of previously-acquired images that must be compared to a real-time image obtained from the endoscope.

RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.13/287,883 filed Nov. 2, 2011, which is a divisional application of U.S.Pat. No. 8,218,847 filed Jun. 4, 2009, which claims priority to U.S.Provisional Application Ser. No. 61/059,669 filed Jun. 6, 2008, theentirety of each of which is incorporated herein by reference.

BACKGROUND

Breakthrough technology has emerged which allows the navigation of acatheter tip through a tortuous channel, such as those found in thepulmonary system, to a predetermined target. This technology comparesthe real-time movement of a sensor against a three-dimensional digitalmap of the targeted area of the body (for purposes of explanation, thepulmonary airways of the lungs will be used hereinafter, though oneskilled in the art will realize the present invention could be used inany body cavity or system: circulatory, digestive, pulmonary, to name afew).

Such technology is described in U.S. Pat. Nos. 6,188,355; 6,226,543;6,558,333; 6,574,498; 6,593,884; 6,615,155; 6,702,780; 6,711,429;6,833,814; 6,947,788; and 6,996,430, all to Gilboa or Gilboa et al.; andU.S. Published Applications Pub. Nos. 2002/0193686; 2003/0074011;2003/0216639; 2004/0249267 to either Gilboa or Gilboa et al. All ofthese references are incorporated herein in their entireties.

Using this technology begins with recording a plurality of images of theapplicable portion of the patient, for example, the lungs. These imagesare often recorded using CT technology. CT images are two-dimensionalslices of a portion of the patient. After taking several, parallelimages, the images may be “assembled” by a computer to form athree-dimensional model, or “CT volume” of the lungs.

The CT volume is used during the procedure as a map to the target. Thephysician navigates a steerable probe that has a trackable sensor at itsdistal tip. The sensor provides the system with a real-time image of itslocation. However, because the image of the sensor location appears as avector on the screen, the image has no context without superimposing theCT volume over the image provided by the sensor. The act ofsuperimposing the CT volume and the sensor image is known as“registration.”

Sensor Probe-Based Registration Methods

There are various registration methods, some of which are described inthe aforementioned references, and utilize a probe with a trackablesensor, as described above. For example, point registration involvesselecting a plurality of points, typically identifiable anatomicallandmarks, inside the lung from the CT volume and then using the sensor(with the help of an endoscope) and “clicking” on each of thecorresponding landmarks in the lung. Clicking on the landmarks refers toactivating a record feature on the sensor that signifies theregistration point should be recorded. The recorded points are thenaligned with the points in the CT volume, such that registration isachieved. This method works well for initial registration in the centralarea but as the sensor is navigated to the distal portions of the lungs,the registration becomes less accurate as the distal airways aresmaller. Also, the point registration method matches a “snapshot”location of the landmarks to another “snapshot” (CT volume) of thelungs. Each snapshot is taken at different times and, potentially, atdifferent points in the breathing cycle. Due to the dynamic nature ofthe lungs, the shape of the lungs during the CT scan is likely not thesame as the shape of those same lungs during the procedure. Moreover,because the physician is “clicking” on the landmarks over the course ofseveral breathing cycles, it is up to the physician to approximate thetiming of his clicking so that it roughly matches the point in thebreathing cycle at which the CT scan was taken. This leaves much roomfor error. Finally, it is time consuming for the physician to maneuverthe sensor tip to the various landmarks.

Another example of a registration method utilizing a trackable sensorinvolves recording a segment of an airway and shape-match that segmentto a corresponding segment in the CT volume. This method of registrationsuffers similar setbacks to the point registration method, though it canbe used in more distal airways because an endoscope is not required. Theregistration should be conducted more than once to keep the registrationupdated. It may be inconvenient or otherwise undesirable to requireadditional registration steps from a physician. Additionally, thismethod requires that a good image exists in the CT volume for any givenairway occupied by the sensor. If for example, the CT scan resulted inan airway shadowed by a blood vessel, for example, the registration willsuffer because the shape data on that airway is compromised.

Another registration method tailored for trackable sensors is known as“Adaptive Navigation” and was developed and described in U.S. PublishedApplication 2008/0118135 to Averbuch et al., incorporated by referenceherein in its entirety. This registration technique operates on theassumption that the sensor remains in the airways at all times. Theposition of the sensor is recorded as the sensor is advanced, thusproviding a shaped historical path of where the sensor has been. Thisregistration method requires the development of a computer-generated andautomatically or manually segmented “Bronchial Tree” (BT). The shape ofthe historical path is matched to a corresponding shape in the BT.

Segmenting the BT involves converting the CT volume into a series ofdigitally-identified branches to develop, or “grow,” a virtual model ofthe lungs. Automatic segmentation works well on the well-defined, largerairways and smaller airways that were imaged well in the CT scans.However, as the airways get smaller, the CT scan gets “noisier” andmakes continued automatic segmentation inaccurate. Noise results frompoor image quality, small airways, or airways that are shadowed by otherfeatures such as blood vessels. Noise can cause the automaticsegmentation process to generate false branches and/or loops—airwaysthat rejoin, an occurrence not found in the actual lungs.

Another registration method is herein referred to as “feature-basedregistration.” When the CT scans are taken, the CT machine records eachimage as a plurality of pixels. When the various scans are assembledtogether to form a CT volume, voxels (volumetric pixels) appear and canbe defined as volume elements, representing values on a regular grid inthree dimensional space. Each of the voxels is assigned a number basedon the tissue density Housefield number. This density value can beassociated with gray level or color using well known window-levelingtechniques.

The sensing volume of the electromagnetic field of the sensor system isalso voxelized by digitizing it into voxels of a specific sizecompatible with the CT volume. Each voxel visited by the sensor can beassigned a value that correlates to the frequency with which that voxelis visited by the sensor. The densities of the voxels in the CT volumeare adjusted according to these values, thereby creating clouds ofvoxels in the CT volume having varying densities. These voxels clouds orclusters thus match the interior anatomical features of the lungs.

By using a voxel-based approach, registration is actually accomplishedby comparing anatomical cavity features to cavity voxels, as opposed toanatomical shapes or locations to structure shapes or locations. Anadvantage of this approach is that air-filled cavities are of apredictable range of densities.

Image-Based Registration Methods

Some registration methods are used with systems that use a bronchoscopewithout a trackable sensor. One of these registration methods comparesan image taken by a video camera to a virtual model of the airways. Thevirtual model includes surfaces, reflections and shadows. This methodwhile herein be referred to as “virtual surface matching.” A virtualcamera is established to generate a viewpoint and a virtual light sourceis used to provide the reflections, shadows, and surface texture. Thevirtual camera and light source are matched to the actual video cameraand light source so that an “apples to apples” comparison can beperformed. Essentially, the virtual model is a library of thousands ofcomputer-generated images of the lungs, from various viewpoints. Hence,the image taken by the video camera is compared against this largelibrary, in the same way a fingerprint is lifted from a crime scene andcompared against a large database of fingerprint images. Once the matchis found, the camera is determined to be where the “virtual camera” waswhen the computer image was generated.

One problem with this method is that each time the camera moves, as itis being advanced toward the target, the images recorded by the cameraare compared against the large library of computer generated images.This is time consuming and places a strain on the computer resources. Italso presents the risk that there may be more than onecomputer-generated image that closely matches the actual image. Forexample, if the video camera is up against an airway wall, there may notbe much on the image to distinguish it from other similar computergenerated images of walls.

Another problem is lack of tracking Without a sensor, there is norecorded history. Hence, even though the camera is moving and beingregistered, as soon as the camera encounters an area that matches morethan one computer generated image, the registration is lost. The systemhas no capacity for “tracking” the movement of the camera. In otherwords, the system does not look at the previous matches to deduce whichof the possible images is likely to be the correct one.

Yet another bronchoscope registration method involves terrain orskeletal surface-matching. The virtual model of the lungs is left in askeletal format, rather than filling the contours in with surfaces andreflections. This saves on initial processing time. As video images arecaptured of the actual lungs, they are converted into skeletal, digitalimages. The “real” skeletal images are then matched against the virtualskeletal images. This method requires more processing of the videoimages than the previously described “virtual surface geometerymatching” method but the matching steps are accomplished much morequickly because each of the virtual images is smaller in terms of data.Like the virtual surface matching method, this method present the riskthat there may be more than one computer-generated image that closelymatches the acquired image, such as when the camera is pointing at awall.

Each of the aforementioned registration methods has advantages anddisadvantages over the others. Generally, the methods using trackablesensors are more accurate than the image-based methods. Moreparticularly, the methods using trackable sensors are more accurate“globally,” that is, they are more accurate when it comes to indicatingthe present position on a scan of the entire lungs. Image-based methods,on the other hand, can be more accurate “locally,” that is, they can bemore accurate relative to a small area, if conditions are optimal. Thus,it would be advantageous to introduce a hybrid method that utilizes theadvantages of all of the aforementioned methods.

SUMMARY

The present invention provides several new or improved registrationmethods. Additionally, the present invention describes a concept wherebya most accurate registration is determined and utilized at any giventime during a procedure, thereby utilizing the advantages of all of theaforementioned registration methods.

More specifically, one aspect of the present invention provides a methodof registering real-time sensor location data to previously acquiredimages of a branched network of body lumens selected from a plurality ofpreviously acquired images stored in a memory of a computing device.This method involves placing a probe containing a sensor at a distal endthereof into a branched network of body lumens in a patient; performingan initial registration between a real-time sensor location and apreviously acquired image selected from a plurality of previouslyacquired images of said branched network stored in the memory of thecomputing device; receiving data from said sensor to determine anapproximate location of said sensor; using said approximate location ofsaid sensor to create a subgroup of said plurality of images, saidsubgroup containing one or more previously acquired images correspondingto said approximate location; and selecting an image from said subgroupthat most accurately corresponds to said approximate location to updatesaid initial registration using an image-based registration technique.

Placing a probe containing a sensor at a distal end thereof may compriseplacing a probe with a six degree of freedom sensor at a distal endthereof.

Performing an initial registration may comprise viewing a landmarkthrough an endoscope; using data from said sensor to project a beam froma tip of said probe to said landmark; displaying said beam on a monitor;calculating and recording coordinates of said beam location on saidlandmark; and using said coordinates as a registration point.

Receiving data from said sensor to determine a proximate location ofsaid sensor may comprise receiving six degree of freedom data from saidsensor.

Placing a probe containing a sensor at a distal end thereof into abranched network of body lumens may comprise placing a bronchoscopecontaining a sensor at a distal end thereof into said branched networkof body lumens.

Selecting an image from said subgroup that most accurately correspondsto said approximate location to update said initial registration usingan image-based registration technique may comprise selecting an imagefrom said subgroup that most closely matches an image being viewedthrough said bronchoscope.

Performing an initial registration between a real-time sensor locationand a previously acquired image selected a plurality of previouslyacquired images of said branched network may comprise performing aninitial registration using a 4D registration technique.

Performing an initial registration using a 4D registration technique maycomprise: recording an image of a landmark as it moves through at leastone breathing cycle; recording concurrently a position of said sensor;recording concurrently positions of patient sensors, said patient sensorattached at various locations on said patient; saving said recordings asa data set for said landmark; and using said data set to correlate saidposition of said sensor to a previously acquired image of said branchednetwork of body lumens.

Another aspect of the present invention provides method of navigating aprobe through a branched network of lumens of a patient comprising:compiling a database of images of said branched network of lumens priorto a navigating procedure; placing a probe containing a sensor at adistal end thereof into said branched network; receiving probe locationdata from said sensor; and using at least said probe location data toselect an image from said database to display to a user navigating saidprobe, said image being representative of a location of said probe.

Compiling a database of images of said branched network of lumens priorto a navigating procedure may comprise compiling a plurality of CTscans.

Placing a probe containing a sensor at a distal end thereof into saidbranched network may comprise placing a probe containing a six degree offreedom sensor at a distal end thereof into said branched network.

Placing a probe containing a sensor at a distal end thereof into saidbranched network may comprise placing an endoscope containing a sensorat a distal end thereof into said branched network.

Receiving probe location data from said sensor may comprise receivingsaid probe's location and orientation from said sensor.

Using at least said probe location data to select an image from saiddatabase to display to a user navigating said probe, said image beingrepresentative of a location of said probe may comprise using said probelocation data to create a subgroup of images from said database, saidsubgroup containing only images that correspond to a vicinity of saidprobe location.

Placing a probe containing a sensor at a distal end thereof into saidbranched network may comprise placing an endoscope containing a sensorat a distal end thereof into said branched network.

Using at least said probe location data to select an image from saiddatabase to display to a user navigating said probe, said image beingrepresentative of a location of said probe further may comprise matchinga real-time image from said endoscope to an image from said subgroup.

Another aspect of the present invention provides a method of registeringreal-time sensor location data to previously acquired images of abranched network of body lumens comprising: placing a probe containing asensor at a distal end thereof in branched network of body lumens in apatient; placing a plurality of patient sensors on said patient;recording an image of an anatomical landmark in said patient as saidlandmark moves through at least one breathing cycle; recordingconcurrently a position of said sensor; recording concurrently positionsof patient sensors, said patient sensor attached at various locations onsaid patient; saving said recordings as a data set for said landmark;and using said data set to correlate said position of said sensor to apreviously acquired image of said branched network of body lumens.

Placing a plurality of patient sensors on said patient may compriseaffixing said plurality of patient sensors to said patient's chest oraffixing a plurality of patient sensors to said branched network.

Using said data set to correlate said position of said sensor to apreviously acquired image of said branched network of body lumens maycomprise using said data set to correlate said position of said sensorto a previously acquired CT image of said branched network of bodylumens.

DETAILED DESCRIPTION

The sensor based and image-based registration methods described aboveare improved upon by combining the advantages of each. Put another way,the image-based registration techniques are improved upon through theuse of a trackable sensor. By monitoring sensor data, an approximateposition of the probe tip is easily determined. The trackable sensor mayinclude a locating implement (typically a transmitter or receiver ofelectromagnetic or acoustic waves) and a location implement orimplements (typically receiver(s) or transmitter(s) of electromagneticor acoustic waves). Such location implement or implements are engaged atone or a plurality of locations along the navigational catheter,typically close to or at a tip thereof and provide location data withrespect to the locating implement or implements. Hence, a database ofvirtual images may be appropriately parsed such that the matchingalgorithm has a significantly reduced number of iterations through whichit must cycle to find a match. The position of the sensor is thus usedas filtering tool to determine which images are locally relevant.

Additionally, the tracking of a tool tip or bronchoscope location willnot be lost in cases of partial or complete obscurity of the video imageor in cases when the bronchoscope is passing a bifurcation while thecamera is pointed away from the bifurcation toward a wall. Due to thetracking capability provided by the trackable sensor, the number ofmatching images will typically be reduced to only one after the outliersare removed. Hence, not only is the matching procedure much quicker, itis also more accurate and less likely to provide incorrect matches.

The image-based registration methods are further improved because theneed for camera calibration is eliminated. Presently, image-basedregistration methods require extensive camera calibration efforts, priorto each procedure, in order to obtain images that can be matched to thevirtual images. Factors such as camera angle and camera distortion mustbe corrected prior to the matching process. Because the use of thetrackable sensor as an additional modality greatly reduces the amount ofdata involved, calibration is much less crucial. In other words, despiteforgoing the calibration step, a match is still likely to be found andaccurate because the number of images the camera image is being comparedto is greatly reduced.

The point registration method described above is also improved by thepresent invention. Recall that presently the point registration methodis comprised of two general steps: 1) finding a predetermined anatomicallandmark using a bronchoscope and 2) “click” on the landmark byadvancing the probe with the trackable sensor until it touches thelandmark, then press a button that records the three-dimensionalcoordinates of the landmark. The present invention obviates the need forthe second step by utilizing the six degree of freedom data provided bythe sensor once the landmark is being viewed through the bronchoscope.This data is used to project a virtual “beam” from the tip of the probeto the target. The virtual beam appears on the monitor and the physicianis then able to record the coordinates of the landmark without actuallyhaving to maneuver the probe into physical contact with the landmark.

The present invention also provides a novel registration method, hereinreferred to as “4D registration.” Rather than clicking on a landmark atan approximated point in the breathing cycle, video registrationinvolves recording an image of a landmark as it moves through at leastone, preferably two or more, breathing cycles. The recording of thelandmark includes a recording of the position of the trackable sensor aswell as the positions of the patient sensors. This way, rather thanacquiring a single data coordinate for each landmark, an entire data setis recorded for each landmark over a period of time and including all ormost of the possible lung positions. This way lung movement may be takeninto account during the registration process. Furthermore, the matchingerror will be minimized if an entire data set is used for each point,rather than a single, three-dimensional coordinate.

For example, assume three registration areas are being monitored. Thepositions of all three are recorded over three separate intervals. Thepatient sensor positions are also being recorded during each of theseintervals as well as the position of the trackable sensor and attachedto each image frame. After the three registration points have beenrecorded over one or more breathing cycles, they are aligned using thepatient sensor positions as an indication of the breathing cycle. Hence,for most of the positions of the patient sensors (extremes excepted),there will be a corresponding position of each of the sensors. Hence,the three intervals during which the recordings were taken are“superimposed” so to speak, as though they were all recordedsimultaneously. Later, during the procedure, the patient sensorpositions are used as an indication of breathing cycle and it can bedetermined at which phase of the breathing cycle the registration ismost accurate. Moreover, this information can be utilized duringnavigation by giving the higher weight to sensor data acquired in aspecific phase of breathing.

Although the invention has been described in terms of particularembodiments and applications, one of ordinary skill in the art, in lightof this teaching, can generate additional embodiments and modificationswithout departing from the spirit of or exceeding the scope of theclaimed invention. Accordingly, it is to be understood that the drawingsand descriptions herein are proffered by way of example to facilitatecomprehension of the invention and should not be construed to limit thescope thereof.

What is claimed is:
 1. A method of registering sensor location data toimages of at least one body lumen, comprising: placing a probe includinga sensor disposed thereon relative to at least one body lumen in apatient; sensing an electromagnetic field generated by anelectromagnetic field generator; generating location data correspondingto a location of the sensor in the electromagnetic field; registeringthe location of the sensor relative to a previously acquired imageselected from a memory of a computing device storing a plurality ofpreviously acquired images of the at least one body lumen based on thelocation data; generating a subgroup of the plurality of previouslyacquired images based on the location data; and updating the registeredlocation of the sensor based on an image selected from the subgroup. 2.The method according to claim 1, further comprising: selecting an imagefrom the subgroup based on the location of the sensor.
 3. The methodaccording to claim 1, further comprising: selecting an image from thesubgroup that most accurately corresponds to the location of the sensor.4. The method according to claim 1, wherein registering a location ofthe sensor includes performing a sensor probe-based registration.
 5. Themethod according to claim 1, wherein updating the registered location ofthe sensor includes performing an image-based registration.
 6. Themethod according to claim 1, further comprising: projecting a virtualbeam from the probe to a landmark of the at least one body lumen; anddetermining a location of the landmark relative to the at least one bodylumen based on a display of the virtual beam.
 7. The method according toclaim 6, further comprising: registering the location of the landmarkrelative to a previously acquired image selected from the plurality ofpreviously acquired images of the at least one body lumen.
 8. The methodaccording to claim 1, wherein registering a location of the sensorfurther comprises: imaging a landmark of the at least one body lumenduring at least one breathing cycle; determining a position of at leastone patient sensor disposed on the patient; generating datacorresponding to the landmark based on at least one of the image of thelandmark, the determined position of the at least one patient sensor,and the determined location data; and correlating the location of thesensor to a previously acquired image selected from the plurality ofpreviously acquired images of the at least one body lumen based on thegenerated data.
 9. A method of registering sensor location data toimages of at least one body lumen, comprising: placing a probe includinga sensor disposed thereon relative to at least one body lumen in apatient; sensing an electromagnetic field generated by anelectromagnetic field generator; determining a location of the sensorrelative to the at least one body lumen, the location of the sensorcorresponding to a location of the sensor in the electromagnetic field;initially registering the location of the sensor relative to apreviously acquired image selected from a memory of a computing devicestoring a first set of previously acquired images of the at least onebody lumen based on the determined location; generating a second set ofpreviously acquired images from the first set of previously acquiredimages based on approximate location data generated by the sensor;selecting an image from the second set based on a comparison between theapproximate location data and the second set; and updating the initiallyregistered location of the sensor based on an image selected from thesecond set.
 10. The method according to claim 9, wherein the second setof previously acquired images is a subgroup of the first set ofpreviously acquired images.
 11. The method according to claim 9, whereinthe comparison between the approximate location data and the second setis based on the most accurate correspondence between the images of thesecond set and the approximate location data.
 12. The method accordingto claim 9, further comprising: projecting a virtual beam from the probeto a landmark of the at least one body lumen; displaying the virtualbeam on a monitor; and determining a location of the landmark relativeto the at least one body lumen based on the display of the virtual beam.13. The method according to claim 12, further comprising: registeringthe location of the landmark relative to a previously acquired imageselected from the first set of previously acquired images of the atleast one body lumen.
 14. The method according to claim 9, whereinregistering a location of the sensor further comprises: imaging alandmark of the at least one body lumen during at least one breathingcycle; determining a position of at least one patient sensor disposed onthe patient; generating data corresponding to the landmark based on atleast one of the image of the landmark, the determined position of theat least one patient sensor, and the determined location of the sensor;and correlating the location of the sensor to a previously acquiredimage selected from the first set of previously acquired images of theat least one body lumen based on the generated data.
 15. A system forregistering sensor location data to images of at least one body lumen,comprising: a probe configured for placement relative to at least onebody lumen in a patient; at least one sensor disposed on the probe andconfigured to generate location data corresponding to a location of thesensor relative to the at least one body lumen; and a plurality ofpreviously acquired images of the at least one body lumen, the locationof the sensor configured to be initially registered to an image selectedfrom the plurality of previously acquired images and to generate asubgroup of the plurality of previously acquired images, wherein animage of the subgroup is selected based on a comparison between thelocation of the sensor and the images of the subgroup to update theinitial registration of the location of the sensor to the image selectedfrom the plurality of previously acquired images.
 16. The systemaccording to claim 15, wherein the sensor is a six-degree-of-freedomsensor.
 17. The system according to claim 15, wherein the probe is abronchoscope.
 18. The system according to claim 15, wherein the subgroupincludes at least one image corresponding to the location of the sensor.19. The system according to claim 15, wherein the probe is furtherconfigured to project a virtual beam to a landmark of the at least onebody lumen, wherein a location of the landmark relative to the at leastone body lumen is determined based on a display of the virtual beam. 20.The system according to claim 15, wherein the comparison between thelocation of the sensor and the images of the subgroup is based on themost accurate correspondence between the images of subgroup and thelocation of the sensor.